Papers with task-specific fine-tuning
Assessing Logical Puzzle Solving in Large Language Models: Insights from a Minesweeper Case Study (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown remarkable proficiency in language understanding and have been successfully applied to a variety of real-world tasks through task-specific fine-tuning or prompt engineering. |
| Approach: | They propose a task that challenges LLMs to identify the locations of mines based on numerical clues provided by adjacent cells. |
| Outcome: | The proposed task requires an understanding of each cell’s state, discerning spatial relationships between clues and mines, and strategizing actions based on logical deductions drawn from the arrangement of the cells. |
Knowledge extraction from aeronautical messages (NOTAMs) with self-supervised language models for aircraft pilots (2022.naacl-industry)
Copied to clipboard
| Challenge: | During pre-flight briefings, aircraft pilots analyse a long list of NOTAMs . the messages are usually written in the English language, but the phrasing is very special . |
| Approach: | They pretrain language models derived from BERT on circa 1 million unlabeled NOTAMs . they reuse the learnt representations on three downstream tasks valuable for pilots - criticality prediction, named entity recognition and translation into a structured language called Airlang. |
| Outcome: | The proposed language model can be used on criticality prediction, named entity recognition and translation into a structured language called Airlang. |
TADA: Efficient Task-Agnostic Domain Adaptation for Transformers (2023.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained transformer-based language models are limited in their expressiveness and domain knowledge. |
| Approach: | They propose a task-agnostic domain adaptation method which is modular, parameter-efficient, and data-efficient. |
| Outcome: | The proposed method is efficient and modular, parameter-efficient, and data-efficient. |
PromptDA: Label-guided Data Augmentation for Prompt-based Few Shot Learners (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing studies on prompt-based few-shot tuning focus on deriving proper label words with a verbalizer or generating prompt templates to elicit semantics from PLMs. |
| Approach: | They propose a framework that leverages label semantics for prompt-based tuning. |
| Outcome: | The proposed framework improves on few-shot text classification tasks by leveraging label semantics and data augmentation. |
Swift Cross-Dataset Pruning: Enhancing Fine-Tuning Efficiency in Natural Language Understanding (2025.coling-main)
Copied to clipboard
| Challenge: | Current approaches for fine-tuning datasets rely on expensive sample ranking processes . data set pruning aims to select a subset of a dataset for efficient model training . |
| Approach: | They propose a method that uses TF-IDF embeddings with geometric median to rapidly evaluate sample importance. |
| Outcome: | The proposed method significantly reduces training and storage costs while maintaining model effectiveness. |
SGA-MCTS: Decoupling Planning from Execution via Training-Free Atomic Experience Retrieval (2026.findings-acl)
Copied to clipboard
| Challenge: | a new framework casts LLM planning as non-parametric retrieval, but high latency of inference-time search and supervised fine-tuning are limitations. |
| Approach: | They propose a framework that casts LLM planning as non-parametric retrieval . they leverage Monte Carlo Tree Search to explore the solution space . |
| Outcome: | Empirical results show that SGA-MCTS can match the performance of SOTA systems without task-specific fine-tuning. |
Gradient Ascent Post-training Enhances Language Model Generalization (2023.acl-short)
Copied to clipboard
| Challenge: | Recent studies show that language models can perform diverse downstream NLP tasks in a zero-shot manner without any gradient updates. |
| Approach: | They show that updating pretrained LMs with Gradient Ascent Post-training enhances zero-shot generalization capabilities without any task-specific fine-tuning. |
| Outcome: | The proposed method can be comparable to 2-3x times larger LMs across 12 different NLP tasks. |
AIDE: Attribute-Guided MultI-Hop Data Expansion for Data Scarcity in Task-Specific Fine-tuning (2025.acl-industry)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning large language models for specific tasks require extensive seed datasets or struggle to balance task relevance and data diversity. |
| Approach: | They propose a data synthesis framework that uses a multi-hop process to expand very few seed data points while ensuring data diversity and task relevance. |
| Outcome: | The proposed framework outperforms state-of-the-art methods in task-specific fine-tuning by over 30%. |
How Much Do Encoder Models Know About Word Senses? (2025.acl-long)
Copied to clipboard
| Challenge: | Word Sense Disambiguation (WSD) is a key task in Natural Language Processing (NLP) however, how well these models inherently disambiguate word senses remains uncertain. |
| Approach: | They evaluate several encoder-only PLMs across WordNet and ODE sense inventories to evaluate their ability to separate word senses without any task-specific fine-tuning. |
| Outcome: | The proposed model outperforms output layer on WordNet and ODE sense inventories by 15 percentage points. |
SPECTER: Document-level Representation Learning using Citation-informed Transformers (2020.acl-main)
Copied to clipboard
| Challenge: | Recent Transformer language models do not leverage information on inter-document relatedness, which limits their document-level representation power. |
| Approach: | They propose a method to generate document-level embeddings using citation graphs. |
| Outcome: | The proposed method outperforms baselines on document-level tasks. |
On the Effectiveness of Sentence Encoding for Intent Detection Meta-Learning (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent studies on few-shot intent detection have attempted to formulate the task as a meta-learning problem. |
| Approach: | They propose to modify a few-shot intent detection task to produce a non-trivially strong performance without further domain-specific adaptation. |
| Outcome: | The proposed model improves on the prototypical network variants with task-specific fine-tuning. |
Learn What Is Possible, Then Choose What Is Best: Disentangling One-To-Many Relations in Language Through Text-based Games (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Language models pre-trained on large self-supervised corpora, followed by task-specific fine-tuning has become the dominant paradigm in NLP. |
| Approach: | They propose to train language models pre-trained on large self-supervised corpora, followed by task-specific fine-tuning on the target domain. |
| Outcome: | The proposed model improves on the previous state-of-the-art model on the Jericho Walkthroughs dataset by 49%. |
Speech-based Slot Filling using Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have shown an unprecedented ability across various language tasks. |
| Approach: | They propose to use prompts and LoRA fine-tuning to improve slot filling robustness . they propose a linearised knowledge injection scheme to integrate dynamic external knowledge into LLMs. |
| Outcome: | The proposed model improves slot filling with noisy ASR transcriptions with 6.7% and 17.6% absolute SLU-F1 improvements compared to a fully fine-tuned Flan-T5-XL model. |
On the Impact of Fine-Tuning on Chain-of-Thought Reasoning (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models have emerged as powerful tools for general intelligence, showcasing advanced natural language processing capabilities. |
| Approach: | They propose to use supervised fine-tuning and Quantized Low-Rank Adapters to improve LLMs' task-specific performance to address privacy and safety risks. |
| Outcome: | The proposed model improves the accuracy of the chain-of-thought reasonings across four datasets and demonstrates that the faithfulness of CoT reasoning decreases. |
Exploring and Mitigating Shortcut Learning for Generative Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent large language models (LLMs) have incredible instruction-following capabilities while maintaining strong task completion ability. |
| Approach: | They propose a framework to encourage LLMs to Forget Spurious correlations and Learn from In-context information. |
| Outcome: | The proposed framework can mitigate shortcut learning by forging spurious correlations and learning from in-context information. |
Pre-Training BERT on Domain Resources for Short Answer Grading (D19-1)
Copied to clipboard
| Challenge: | Pre-trained contextualized representations have achieved state-of-the-art results on multiple downstream NLP tasks by fine-tuning with task-specific data. |
| Approach: | They propose to augment domain-specific data by using labeled short answering grading data for further enhancement of the pre-trained language model. |
| Outcome: | The proposed model can be enhanced by augmenting data from domain-specific resources like textbooks and labeled short answering grading data. |
Distilling Relation Embeddings from Pretrained Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models capture a surprisingly rich amount of lexical knowledge, but it is unclear to what extent relation embeddings can be used to encode relational knowledge. |
| Approach: | They found that word vector differences capture lexical relations . relationship embeddings can be used to encode relational knowledge . |
| Outcome: | The results are highly competitive on analogy (unsupervised) and relation classification (supervised) benchmarks, even without any task-specific fine-tuning. |
Mixture-of-Skills: Learning to Optimize Data Usage for Fine-Tuning Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models are fine-tuned on diverse datasets to develop a range of skills . each skill has unique characteristics, and datasets are heterogeneous and imbalanced . a general, model-agnostic, reinforcement learning framework is proposed to optimize data usage . |
| Approach: | They propose a general, model-agnostic, reinforcement learning framework that optimizes data usage automatically during the fine-tuning process. |
| Outcome: | The proposed framework optimizes data usage automatically during the fine-tuning process. |
A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignment (2025.acl-long)
Copied to clipboard
Jean-Philippe Corbeil, Amin Dada, Jean-Michel Attendu, Asma Ben Abacha, Alessandro Sordoni, Lucas Caccia, Francois Beaulieu, Thomas Lin, Jens Kleesiek, Paul Vozila
| Challenge: | Large language models such as GPT-4 have limited their deployment in clinical settings . a novel framework for adapting SLMs into high-performing clinical models is needed . |
| Approach: | They propose a framework for adapting large language models into high-performing clinical models . they pre-instruct experts on relevant medical and clinical corpora and model merging . |
| Outcome: | The proposed framework outperforms the existing model on the CLUE+ benchmark on medical entities and radiology reports. |
Contra4: Evaluating Contrastive Cross-Modal Reasoning in Audio, Video, Image, and 3D (2025.emnlp-main)
Copied to clipboard
Artemis Panagopoulou, Le Xue, Honglu Zhou, Silvio Savarese, Ran Xu, Caiming Xiong, Chris Callison-Burch, Mark Yatskar, Juan Carlos Niebles
| Challenge: | a recent study shows that multimodal models can reason across multiple modalities . a limited number of models are able to reason across a variety of inputs . |
| Approach: | They propose a dataset for contrastive cross-modal reasoning across four modalities . they use human annotations and a mixture-of-models round-trip-consistency filter . |
| Outcome: | a new model evaluates models on multiple modalities to determine which one best answers a natural language prompt . the model must select the one that best satisfies the query and then fine-tune it . state-of-the-art models still achieve only 56% accuracy overall and 42% in four-modal settings . |
Reusable Experiences: Latent Routing and Modular Composition in LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches represent accumulated experience as explicit textual artifacts in prompts or implicitly within model weights via fine-tuning. Existing methods are limited by context windows and cannot internalize knowledge. |
| Approach: | They propose a framework that treats latent experiences as fundamental units for LLM specialization. |
| Outcome: | Experiments on multi-task NLP benchmarks show that this approach outperforms standard fine-tuning, yielding improved generalization through flexible skill reuse. |
InfiAgent: An Infinite-Horizon Framework for General-Purpose Autonomous Agents (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing LLMs break down on long-horizon tasks due to unbounded context growth and accumulated errors. |
| Approach: | They propose a framework that externalizes persistent state into a file-centric state abstraction and keeps the agent’s reasoning context strictly bounded regardless of task duration. |
| Outcome: | Experiments on DeepResearch and an 80-paper literature review show that the proposed framework maintains higher long-horizon coverage than baseline models without task-specific fine-tuning. |
ViLL-E: Video LLM Embeddings for Retrieval (2026.acl-long)
Copied to clipboard
| Challenge: | Video Large Language Models excel at video understanding tasks where outputs are textual . however, they underperform specialized embedding-based models in Retrieval tasks . |
| Approach: | They propose a video-LLM-based model with an embedding generation mechanism that allows the model to "think longer" for complex videos and stop early for easy ones. |
| Outcome: | The proposed model outperforms specialized embedding-based models in video understanding tasks while remaining competitive on VideoQA tasks. |